The advent of virtualization technologies for commodity hardware has provided benefits with respect to managing large-scale computing resources for many clients with diverse needs, allowing various computing resources to be efficiently and securely shared by multiple clients. For example, virtualization technologies may allow a single physical computing machine to be shared among multiple users by providing each user with one or more virtual machines hosted by the single physical computing machine, with each such virtual machine being a software simulation acting as a distinct logical computing system that provides users with the illusion that they are the sole operators and administrators of a given hardware computing resource, while also providing application isolation and security among the various virtual machines. As another example, virtualization technologies may allow data storage hardware to be shared among multiple users by providing each user with a virtualized data store (e.g., a virtualized database) which may be distributed across multiple data storage devices or storage nodes, with each such virtualized data store acting as a distinct logical data store that provides users with the illusion that they are the sole operators and administrators of the data storage resource.
As the scale and scope of network-based applications and network-based services such as virtualized or cloud computing services have increased, network-based applications or services may include hundreds or even thousands of hardware or software nodes to which client traffic from many clients of the service or application may need to be load balanced. In an example data center or network implementing a large-scale service or application, multiple load balancers in a load balancer layer may front a fleet of nodes, with each load balancer fronting a subset of the nodes. Client connections to the service or application may be pseudorandomly distributed among the load balancers in the load balancer layer, for example by a Domain Name Server (DNS) accessible by the clients that provides the IP addresses of the load balancers to the clients upon request. Connections from clients (e.g., Transmission Control Protocol (TCP) connections) may thus be established to particular load balancers; the load balancer to which particular client connections are established may distribute client traffic on those connections among its respective nodes according to a load balancing technique.
Load balancers are typically single, dedicated devices that include multiple network interface controllers (NICs), for example eight NICs, with some of the NICs handling inbound traffic from/outbound traffic to clients and the other NICs handling outbound traffic from/inbound traffic to the nodes that are being load balanced. Bandwidth or throughput on conventional load balancers is typically in the range of 40 Gigabits per second (Gbps) on the client side and 40 Gbps on the server side. Load balancers typically use techniques such as max connections (or max conns), round robin, and/or least connections (least conns) applied to data collected from the host devices to select which node will handle a connection. In addition, load balancers typically serve as proxies to the nodes that they front and thus terminate connections from the clients and send the client traffic to the nodes on connections established between the nodes and the load balancer. Thus, a node and a client typically do not communicate over a direct connection when using load balancers.
While embodiments are described herein by way of example for several embodiments and illustrative drawings, those skilled in the art will recognize that embodiments are not limited to the embodiments or drawings described. It should be understood, that the drawings and detailed description thereto are not intended to limit embodiments to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope as defined by the appended claims. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description or the claims. As used throughout this application, the word “may” is used in a permissive sense (i.e., meaning having the potential to), rather than the mandatory sense (i.e., meaning must). Similarly, the words “include”, “including”, and “includes” mean including, but not limited to.